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Overview 


►  Introduction 

►  System  Architecture 

►  Approach  and  Challenges 

►  Unscripted  Test 

►  Day-in-the-Life  Test 

►  Incremental  Test 

►  Risk  Management:  Quality  vs.  Progress 

►  The  Contract  and  Award  Fee  Plan 

►  Final  Thoughts 


Introduction 

“Government  Beta” 
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The  Beta  Effect 


►  If  you  want  to  see  your  software  do  what  it 
was  designed  to  do,  put  it  in  the  hands  of  its 
users  (literally  and  figuratively) 


Can't  you  do 
anything  right? 


The  Process  of  Development 


The  Incremental  Effect 


The  Government’s  Role 


►  Develop  Requirements 

►  Develop  Request  for  Proposal 

►  Award  Contract 

►  Oversee  Execution 

►  Manage  Risk 

►  Manage  Contract  Modifications 

►  Manage  Award  Fee/Incentives 

►  Accept  Product 

ment 

*Best  Opportunities  to  Influence  Outcome 
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Oversee  Sustain 


The  Pursuit  of  Quality/Suitability 


Completeness, 
Accuracy  &  Language 


Improvements  not  made 

Government 


7 


System  Architecture 


I  COVER 
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Our  System  -  Advanced  EHF 


Legacy  Satellite  (Milstar) 


New  Satellite  (AEHF) 


New  Satellite  (AEHF) 


Mission  Control  Segment  (MCS) 


Mission  Planning  Element  (MPE) 

►  Responsible  for  planning,  scheduling,  execution, 
data,  and  monitoring  satellite  and  terminal 
resources 

►  Multiple  users  (1 00’s)  greatly  varying  in 
knowledge,  skills  and  mission 

►  Extensive  reuse  of  software  from  a  previous 
system 

►  High  complexity  due  to  payload/terminal  designs 

•  1 00’s  of  screens,  interdependent  data 

•  >  1.5M  SLOC  in  C/C+  + 

•  Software  is  classified  SECRET 


MPE  High  Level  Conops 


Hundreds  of  MPSS 
Distributed  Planning 
-  From  National  Command 
Authorities,  through  all 
Echelons,  to  Field  Users 
One  MPE  Application  Set 
Coordination  Through  Data 
Distribution 


Plan  Strategic  Networks 
Develop  UCC  Apportionments 
Distribute  Apportionment  to  UCCs 
Process  Resource  Utilization  Data 
Analyze  Resource  Utilization  Data 


Sub-Apportion  Resources 
Distribute  to  UCCs 
Process  Resource  Utilization  Data 
Analyze  Resource  Utilization  Data 


Navy 

Component 


Air  Force 
Component 


Marine 

Component 


\i»  i  A 


aEvi'  **,  i 


Army 

Component 

Sub-Fence  Resources 
Distribute  to  Corps 
Develop  Detailed  Plan 
Generate  Terminal  DB  and 
Execution  Plan  Data 
Analyze  Resource  Utilization  Data 


Antenna  Pointing/ 


Develop  Detailed  Plan 

Generate  Terminal  DB  and  Execution 

Plan  Data 

Analyze  Resource  Utilization  Data 


Antenna  Controllers/ 
Terminals 

Re-point  Antennas 
Establish  Fences 
Activate  Nets 

Analyze  Resource  Utilization  Data 


Many  Users  /  Many  Different  Missions 


MPE  Mission  Executables 


MPE  Element  Spec 

Component  Requirement  Specifcations 

MPE  Mega/Major  Functions 

#  CRS 
Req 

MPE  Mega-Executables 

Count 

MPE  Planning  (PMP) 

301 

Apportionment  and  Planning 

396 

Crypto  Management 

112 

MPE  Terminal  Support  (PMT) 

136 

Terminal  Data  Generation 

139 

Terminal  Control 

224 

MPE  Resource  Monitoring  (PMR) 

102 

Resource  Monitoring 

288 

MPE  Guards  (PMG) 

16 

Guards 

64 

MPE  IP  Flow  Control  (PMI) 

15 

IP  Flow  Control 

2 

Total 

570 

General  MPE 

135 

MPE  CRS  Total 

1360| 

Common-Services 

Count 

Common  Mega  -Service 

Kernel/Operating  Sys  Services  (CKL) 

68 

Kernel/Operating  System  Services 

49 

Database  Services  (CDS) 

5 

Database  Services 

1 

Security  Services  (CSS) 

4 

Security  Services 

12 

(Archive  Services  (CAS) 

6 

Archive 

2 

Total 

83 

General  COE 

1 

Common  Total 

65 

Terminal 


Many  Requirements  Spanning  Multiple  Executables 


Database  Business  Rules 


MPE  Software  Architecture 
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(DLL’s) 

LL 

LL 

DBCM 

ODBC 

Database 

Layer 

Oracle 

OS 

Layer 


C/C++ 

MFC 

SQL 


C/C++ 

SQL 


Business  Rules  Span  Multiple  Layers 


IS 


How  should  you  test 
a  system  like  this? 


Approach  and 
Challenges 

It’s  Not  the  Same  as  Last 
Time 
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MPE  Challenges 


►  Organizational  Bias 

•  Bias  towards  doing  the  way  we  did  it  last  time 

•  Desire  to  reuse  previous  standard  test  procedures 

►  Requirements  Definition 

•  MPE  expected  to  be  “all  things  for  all  users” 

•  Requirements  are  never  perfect 

•  Evolving  ConOps  (and  requirements  interpretation) 

►  Requirements  Verification 

•  Requirements  sold-off  at  lowest  applicable  level  of  testing 
•  Unintended  Regression  Danger 

•  Significant  effort  for  testing  of  scenarios  and  DR  removal 
remain  after  many  requirements  already  “sold  off’ 

►  Schedule  Pressure 


Focus  getting  cases  completed,  not  straying  from  the  script 
Testing  this  system  difficult  and  time  consuming 


Scripted  Testing  Classic  Problems 


►  Test  Scripts  Are  Not  Correct 

•  Can’t  always  capture  the  steps  needed  to  prove  the 
software  was  working  correctly  all  the  time 

►  Inattentional  Blindness 

•  Blind  to  errors  that  are  not  part 
of  the  test  case 

►  Inaccurate  Perceptions 

•  Perception  that  completing  test  cases 
is,  by  itself,  an  accurate  measure  of 
progress  -  it  is  not 

•  Perception  that  if  you  have  passed 
the  test  cases  the  software  is  ready 


Observations  and  Conclusions 


►  Overall 

•  Product  acceptance  based  on  requirement  verification 
alone  not  the  correct  strategy  for  a  product  as  complex 
and  flexible  as  MPE 


►  Specific 


•  Formal  tests  typically  use  small  input  data  sets 

•  Testers  stick  to  script,  many  operational  logic  paths  left 
untested,  lack  of  operational  flavor  in  test 

•  Insufficient  system-level  and  operational  environment 
experience  among  testers 

•  Again,  many  operational  logic  paths  not  tested 

•  Insufficient  peer  reviews 

Defect  escapes  from  design  inspections,  code  inspections 


Insufficient  integration  tests 
Lack  of  diversity  in  testing 


Test  Process  Improvements 


Challenge 

Corrective  Action 

Lack  of  operational  oriented 
testing 

•  Added  Independent  Pre-Ops  Test 
program 

Peer  reviews  and  integration  and 
formal  test  approach  insufficient 

•  Added  Formal  Integration  Test  (FIT) 

•  Increased  Peer  Reviews 

Requirements  verification  alone 
insufficient 

•  SPO  acceptance  after  successfully 
completing  government  provided 
Day-in-the-Life  (DITL)  test 

Unscripted  Test 

Unscripted,  not  Unplanned 
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Classic  Software  Lifecycle  Model 


Operational 

Requirements 


System 

Requirements 


System 
Design 

Software 

Requirements 

Software 
System  Design 


.Review 


Software 
Detailed  Design 

Code 


Operational 

Acceptance 

Test 

Government 
Developmental 
&  System  Test 


SW  /  HW 

Integration  &Test 


SW  Item 
Test 

SW  Subsystem 
&  Integration 
Test 


Integration 

Test 

Unit  Test 


IEEE  12207 


-  PM,  CM,  SQA 


MPE  Equivalent  Lifecycle  Model 


■■ 


Formal  Test  Program 


Requirements  Verification 


Test  Cases  Completed 
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Unscripted  Testing 

►  Form  of  quality  checking  that  does  not  rely  on 
test  scripts.  A  tester  is  let  loose  on  the  system 
and  encouraged  to  report  all  issues. 

►  How  did  we  incorporate  unscripted  test  in  AEHF? 

•  Formal  Integration  Test  (FIT) 

-  Contractor  led  mega-level  integration  test,  freedom  to  test 
wide  range  of  values 

•  Pre-Ops  Test 

-  Independent  test  team  focused  on  mission-based  unscripted 
testing 

•  Day-in-the-Life  Test 

-  Three  day  test  written  by  government  for  contractor  to 
assess  product  operability  and  effectiveness 


Formal  Integration  Test  (FIT) 


►  Developer-led  test 

►  Bridges  gap  between  requirements  testing  and  unscripted 
testing.  Specifies  fields  to  test,  but  freedom  to  test  wide 
range  of  values. 

►  Matrices  developed  by  examining  fields  and  subset  of 
variables  to  be  tested 


►  Sequence  of  parameterized  steps  associated  with  a  particular 
function.  Variable  of  Interest  x  Range  of  Interest 


Number  of 

Total  Number  of 

Tester 

Deep  Dive 

Testing 

Fields  on  Display 

Range  of  Interest 

Items  to  Test 

Items  Tested 

Items  Tested 

Initials 

Deep  Dive 

section 

complete 

Active  Term  Status  Tab: 

- 

Yes 

User  Defined  Filter 

none 

0 

0 

Yes 

Start  Date/Time 

none 

0 

0 

Yes 

Stop  Date/Time 

none 

0 

0 

Yes 

Origin 

none 

0 

0 

Yes 

Source  Id 

1,  2,  term  Id 

3 

1 

1 

TDG,  SHT 

RM-4,  RM-3 

4-3 

No 

1  in  US  Range 

2048 

TDG 

RM-4 

4-3 

Term  Id 

1  in  IP  Range 

2 

1 

4000 

SHT 

RM-3 

No 

1  L/M  capable, 

1  L/M/X  capable. 

NMT SHORE 

TDG 

RM-4 

4-3 

Term  Type 

1  XDR  only  capable 

3 

1 

SMART-T 

SHT 

RM-3 

No 

Owner 

US  parent  org, 

US  sub  org 

2 

1 

CENTCOM 

ARMY 

TDG 

SHT 

RM-4 

RM-3 

4-3 

No 

Event  Time 

none 

0 

0 

Yes 

Term  Status  Results 

Tab: 

_ 

_ 

Yes 

Event  Time 

none 

0 

0 

Yes 

Origin 

none 

0 

0 

Yes 

Source  Id 

1,  2,  term  Id 

3 

1 

1 

TDG 

RM-4 

4-3 

No 

1  in  US  Range 

2048 

TDG 

RM-4 

4-3 

Term  Id 

1  in  IP  Range 

2 

1 

4000 

SHT 

RM-3 

No 

1  L/M  capable, 

1  L/M/X  capable, 

NMT  SHORE 

TDG 

RM-4 

4-3 

Term  Type 

1  XDR  only  capable 

3 

1 

SMART-T 

SHT 

RM-3 

No 

US  parent  org, 

CENTCOM 

TDG 

RM-4 

4-3 

Owner 

US  sub  org 

2 

1 

SMART-T 

SHT 

RM-3 

No 

any  1  M*  sat, 

3 

TDG 

RM-4 

4-3 

Sat  Id 

any  1  AEHF 

2 

1 

9 

SHT 

RM-3 

No 
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Why  Formal  Integration  Testing? 

►  Product  experienced  latent  defect  discovery  in  deployment 
and  acceptance  testing 

►  Process  analysis  highlighted  need  to  enhance  procedure 
driven  requirement-based  testing  with  scripted  free-form 
combinatorial  testing 


Observed  Defect  Density  by  Phase 


System  Design  CUT  MLT/SLT  ELT  Deployment 

Design 


Actual:  High  (latent)  Defect 
Density  in  Deployment  Phase 


y  _  J 


Finding  the  Right  fit  for  FIT 

►  Formal  Integration  Testing  (FIT)  was  used  across 
three  increments 

►  The  benefit  of  FIT  increased  as  it  moved  left, 
driving  out  problems  earlier  in  the  development 
lifecycle 
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Value  of  Formal  Integration  Testing 


Defects  and  DRs  per  K-ELOC 


DD  CUT  FIT  MLT  ELT  PIT  SLT  Deploy 


•  Reduction  in  cost:  Industry  research  shows  that  problems 
are  easier  (and  cheaper)  to  fix  earlier  in  the  lifecycle 

•  Reduction  in  schedule:  Fewer  problems  =  fewer  test  re¬ 
starts  =  shorter  test  program  duration 

•  Increase  in  confidence:  Higher  quality  product  delivered 
with  fewer  defects  and  fewer  required  workarounds 
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What  is  Pre-Ops  Testing? 


►  Independent  test  team  (non-developers) 

►  Mission-based  unscripted  testing 

►  Mission/domain  experts 


►  Resembles  a 
Beta  Program 


Pre-Ops  Testing 


Strat  Strat 
Planning  1  Planning  2  Crypto 


MODC 


PL,  US /IP 
Appt  Planning  MPE 


International 
Partner  (IP) 
Planning  UCC 


Tactical 
Planning  UCC 


Hand-Holding  vs.  Pointing  in  Right  Direction 


►  Test  Engineer 

Formal  Test  Case 

Create  a  communications  plan  and  add  the 
set  of  service  configurations,  BSMRCA 
Events,  and  Beam  Events  to  that 
communications  plan. 

•  On  the  BSMRCA  Events  Tab,  add  the 
following  Event  to  the  Event  List: 

•  Sat:  10 

•  Beam:  BSMRCA- 1 

•  Duty  Cycle:  75,  25 

•  Date/Time:  Start  time  is  default 
start  of  Communications  Events. 


►  Functional  Experts 

Pre-Ops  Test  Objective 

Plan  for  a  total  force  deployment  (100  terminals, 

40  communication  services) 

•  Plan  for  a  total  force  deployment  (100 
terminals,  40  communication  services)  in  a 
common  Comm  Plan  using  all  Service  types 
(combination  of  LDR/MDR/XDR  to  include 
OTADD  &  Navy  RB  Nets) 


Motivation:  Complete  test  cases  for  I  Motivation:  Find  as  many  defects  as 
verification  on  or  before  schedule  I  possible  (within  schedule) 


Value  of  Pre-Ops  Testing 

►  Mission  Ops-Based  Focus 

•  Ops-like  users  performing  real-world  mission  tasks 

•  Accounts  for  multiple  missions,  users  and  levels  of 
experience  and  knowledge 

►  Similar  to  typical  “beta”  testing 

•  Users  using  product  in  unanticipated  ways 

•  Delivers  more  diversity  in  testing 

•  Uncovers  “user  experience”  issues 

►  Finds  defects  in  development,  not  in 
operations 


Government  Developed 
Day-in-the-Life  Testing 

Put  Your  Money  Where 
Your  Mouth  Is 


BETA  -  The  Value  of  Unscripted  Testing 
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Issue 


►  Requirements  verification  alone  inadequate 
for  proving  operability  and  effectiveness  of 
the  MPE  product 

•  Lack  of  diversity  in  testing 

•  Users  finding  defects  that  should  have  been  found 
earlier 

•  Users  finding  issues  in  real-world  conditions  not 
represented  in  factory  test  environment 


DITL  Objective 

►  The  primary  objective  of  the  DITL  was  to 
assess  the  operability  and  effectiveness  of 
the  MPE  product 

•  Can  it  accomplish  a  real-world  mission  under 
real-world  conditions? 

•  Is  it  ready  for  the  user? 


DITL  Test  Case  Description 


►  Excerpt  from  Segment  Level  Test  Plan 

1.1.1.1  (U)  Test  Case  I7-SLT-S0Q7-003 
(U)  Days-in-the-Life  MPE|Planning 

1.1. 1.1.1  (U)  Test  Case  Description 

(U)  The  objective  of  the  Increment  7  MPE  DITL  is  to  assess  operability  and  effectiveness  of  the 
increment  7  MPE  product.  It  will  be  run  once  all  other  Increment  7  requirements  and  product 
integration  testing  are  completed.  The  test  will  be  operationally  relevant  and  exercise  a  broad 
array  of  MPSS  capabilities  consistent  with  the  Increment  7  functional  baseline.  There  will  be 
no  traditional  test  script  driving  the  activities  at  the  defined  MPSS  stations.  Instead, 
MPSS  operators  execute  a  set  of  Government  provided  tasks  nominally  expected  to 
occur  in  operations.  These  tasks  may  span  multiple  MPSS  User  Roles  and  may  or  may  not 
need  to  occur  sequentially.  Each  task  will  be  sized  appropriately  with  respect  to  nominal 
operational  processing  and  the  total  effort  of  all  tasks  is  not  expected  to  exceed  72  hours  of  test 
conduct. 


The  Contractor  will  request  the  MPE  DITL  test  tasks  after  completing  all  predecessor 
test  cases  to  the  MPE  DITL  and  baselining  all  MPSS  DITL  impacting  DRs. 


(U)  There  are  no  requirements  that  will  be  verified  in  this  test  case 
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DITL  Characteristics 


►  Operational  Configuration 

►  Operational  Inputs/Outputs 

►  Contractor  personnel  as  users 

►  Mission  Driven/Mission  Scenario 

►  Contractor  does  not  know  ahead  of  time  what 
will  be  on  the  test 

►  Passing  DITL  is  an  exit  criteria  for  Segment 
Level  Test 

►  Must  be  able  to  accomplish  the  mission 

►  Operationally  acceptable  workarounds  OK 
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DITL  Timeline 


Timeline 

Event 

-  90  Days 

Baseline  TEM.  Agree  on  starting  DB  and  content,  unique  test 
configuration  requests 

-  30  Days 

Walk  Through  Event.  Multi-Day  Site  Walk-through  of  anticipated 
procedures,  test  threads,  data  capture  methods,  etc. 

-  2  Weeks 

Contractor  Indicates  “Ready  To  Execute” 

-  3  Days 

Contractor  Requests  MPE  DITL  Test  Scenario  and  Government  confirms 
entry  criteria  met  (Successful  TRR) 

-  3  Days 

Government  provides  DITL  Test  scenario  and  supporting  data.  The 
government  and  a  contractor  representative  will  review  the  mission 
planning  scenario  to  ensure  the  missions  are  understood,  in  scope 
and  relevant  to  the  test  objectives. 

+  0  Days 

Execution  of  DITL  scenario  steps  begins.  (End  of  day  tag-ups:  (1 ) 
Contractor,  (2)  User  Representatives) 

+  8  Days 

Execution  completes  after  72  hours  of  test  execution  time  or 
successful  completion  of  steps 

+  1 0  Days 

Complete  Analysis  and  Government  Chief  Software  Engineer  issues 
findings  (Pass/Fail) 

DITL  Operational  Configuration 


St  rat  St rat 

Planning  1  Planning  2  Crypto 


International 
Partner  (IP) 
Planning  UCC 


Tactical 
Planning  UCC 


IP  Flow 
Control  MPE 


MPE  Archive  to  OSSE 


USER  ROLES 


Strati  Crypto  PL,US/IP  Strat2  Crypto  Tactical  IP  Planner  4SOPS 
Appt  Planner  &  DCA*  RM 

Planner 


MPE1  MPE2  MPE  3  MPE4  MPE2 


MPE5  MPE6 

IP  Flow* 


Migration 


Strategic  Planning 
And  Operations 


Nominal 

Planning 


Implementation 
And  Distribution 


MODC1 

MPE3 


IP 

Planning 


Nominal 

Resource 
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DITL  Input  Scenario  Example 


Strategic  Planning 

Task  ID 

Scenario 

Name  ^ 

*  Data  ^ 

^EyaL  Criteria 

✓Output  Product  to  PDF  and 
save  to  directory  (eg. 

I : 'DitLPr  o  du  ct  s'MMYYYY/ 

Time  Est. 

f 

Task  Desc 

V 

ription 

_ y 

^S/IAGE 

In  Endurance M^on  Re-plan. 

fail  Ml ^y  . 

Taki^dSeenshots  of 

Impacted 

S  ervi  ces.-T  erminal  s 
HMI 

Re-plam^^rerminal  s 
and^OTices 
^^^owing  criteria  in 
requirement 

MPE2805. 

Comm  Plan  Summari- 
Report :  Terminal  and  Service 
Configuration  Reports  for 

Li  Impa  cte  d  S  ervi  c  es.'T  erminal  s" 5 

1  hr 

26. 

myy" 

The  foho^fM^fasks  will  be 
perfpm&d'to  verify  functions 

dently ,  but  are  not  > 

^dependent  on  previous  tasks^X 
for  strategic  planning  yy^ 

Input  Da 
Operatic] 
Form? 

i  ! 

ta  in 
>nal 

It 

Move  M4  sat  in  A3 

Reference  Tab  S-S 

Sat  moved  properly 

N/A 

10  min 

V 

- /I 

<Tmage 

Add  A3  in  L»tftraisition 
mode  y^y^ 

Reference  Tab  S-9 

Sat  created  &■ 
payload  (s) 
configured  properly 

Payload  Configuration 
reports 

20  min 

f  1 

Evaluation 

V  J 

IMAGE 

^ figure  crosslinks 

Reference  Tab  S-9 

All  sats  cross-hnked 
successfully 

LDR-MDR’XDR 

Constellation  Configuration 
Reports 

1 5  min 

Generate  A3  Payload  Table 
(using  Payload  Table 

Generation  Application) 

AEHFDCID51 
(Downlink  Channel 
Parameters  Table) 

P/L  Table  file 
generated 

N/A 

5  min 

IMAGE 

Make  A3  changes  in 

Frequency  Planning  Wizard 

Reference  Tab  S- 
10 

Frequency  Planning 
Wizard  completes 

Screenshots  of  Demo d  Map 
and  WCB  Map 

20  min 

r~ 

Created  0 
Products 
Later  Evak 

utput 

for 

cation 

IMAGE 

Save  changes  to  A3  frequency 
plan  as  LtLate  Transition 
Modified"  template 

N/A 

Template  shows  in 
Frequency  Planning 
Template  list 

XDR  Frequency  Planning 

Rep  on 

10  min 

V  J 

Table  1  -  Strategic  Planning  Tasks  ( 9*5  hours) 
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Value  of  DITL 


►  Provided  a  test  gate  based  on  proving  the 
software  could  accomplish  a  real-world  mission 

►  Gave  the  government  leverage  to  assess 
suitability  for  ops  despite  no  formal  suitability 
requirement 

►  Incentivized  the  contractor  to  introduce  new 
forms  of  mission  based  testing  using  real-world 
operators  (Pre-Ops  Testing)  to  ensure  they  could 
get  through  this  gate 

►  Dramatically  improved  the  quality  of  the  product 


Incremental  Test 

Progress,  not  Regress 


GOVERNMENT  BETA  -  The  Value  of  Unscripted  Testing 
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Automate,  Automate,  Automate 


►  A  system  that  is  ridiculously  hard  to  test  is 
equally  ridiculously  hard  to  regression  test 

►  Automated  testing  expedites  verification  and 
protects  against  regression 

►  Some  systems  are  more  difficult  to  automate 
than  others 


Design  for  Test 

►  Implementing  automated  testing  can  be 
simplified/facilitated  through  design 


000 

ooo 


test  states 
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Deliverable  Test  Scripts 


) 


Contract 


- \ 

Extensible  Test  Framework  & 
Core  Test  Scripts 

_ / 


►  Protects  against  regression 

►  Improves  and  expedites  verification 

►  Promotes  good  architecture  and  design 

►  Supports  sustainment  activities 


From  Increments  to  Early  Drops 


Now  they  really 
are  beta  releases 


|  Gov 

JC 


Government 

Testing 


Y 


Government 

Testing 


Y 


Government 

Testing 


Y 


Government 

Acceptance 


Automated 
Regression 
&  Pre-Ops 


Automated 
Regression 
&  Pre-Ops 


Iteration  n 


Iteration  n  +  1 


Iteration  n+2 


V 


Beta  Program 


J 


Risk  Management 

Quality  vs.  Progress 
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Measuring  Progress  vs.  Quality 

►  Progress  or  Quality? 


►  Completed  test  cases  may  not  reflect  quality 

►  Earned  Value  claimed  on  “defects  closed”  rather  than  “defects 
remaining  open”  can  be  a  misleading  indicator  of  progress 


Software  Maturity  Risk 


Initial  Risk  Definition 


delayed  and  the  ability  to  field  the  software  could  be  adversely 
impacted. 

Risk  Areas: 

Unplanned  integration  problems  /  rework 


Burn  Down  Schedule 


*  If  the  software  products  have  a  significant  quantity  of  DRs 

Rf 

10 

■Q 

06 

2007 

2008 

2009 

2010 

2011 

(including  DRs  of  a  high  severity)  and/ora  large  quantity  of 

Q4 

Q1  Q2  Q3  Q4 

Q1  Q2  Q3  Q4 

Q1  Q2  Q3  Q4 

Q1  Q2  Q3  Q4 

Q1  Q2  Q3  Q4 

projected  latent  DRs,  then  the  ability  to  execute  System  Test 

High 

will  be  adversely  impacted,  program  schedules  could  be 

Q 

Mitigation  Plans 


Challenge  to  measure  SW  Reliability 
-  hrs/defect,  defect  density,  defect 
priority,  defect  backlog 


Milestone 


Complete  DR  Assessment 


Weak  SW  Reliability  Detected 


SW  Reliability  Growth  Confirmed 


Strong  SW  Reliability  Growth  Detected 


Risk  Level 

R  5L,5C 
R  5L.5C 
R  4L.5C 
R  3L,4C 


Strong  SW  Reliability  Growth  Confirmed  Y  3L,3C 


Day-ln-The-Life  (DHL)  Completion 


G  2L3C 


Monitor  Inc  5  Maturity  Through  1ST  (6K-2) 


•a  4 

o 

o 

i  3 

0) 

-«  2 


2  3  4  5 

Consequences 


o 

Complete 

o 

Partial 

o 

Future 


Managing  Risk 

►  An  unscripted,  government  developed  test  like 
the  DitL  can  improve  software  quality 

•  Establishes  a  mentality  with  the  contractor  that  the 
software  just  has  to  work! 

•  Less  about  the  metrics,  more  about  functionality  and 
suitability 

►  How  do  we  measure  software  maturity? 

•  Common  understanding  of  how  defects  are 
captured/measured 

•  Normalized  metrics,  Intuitive  front-end  for  viewing  defects 

•  Assuming  a  constant  level  of  test  and  fix  rate,  defect 
backlog  should  be  decreasing 

•  Completing  test  cases  is  not  a  measure  of  quality 

•  Automated  regression 


The  Contract  and  Award 

Fee  Plan 

Where  Rubber  Meets  Road 
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You  Get  What  You  Ask  For 


►  The  contract  and  award  fee  plan  set  the  stage 
for  program  execution  and,  ultimately,  the 
quality  of  the  product  delivered 

►  The  following  performance  criteria  encourage 
schedule  over  quality 

•  Technical  Milestone  Performance 

•  Deliver  a  product  that  meets  requirements 

•  Cost  Performance 

•  Earned  Value  Management  (maintain  good  SPI  &  CPI) 


The  Contract 


Unscripted  Testing 
(FIT,  Pre-Ops) 


Deliverable  Extensible  Test 
Framework  &  Core  Test  Scripts 
(automated  regression) 


Iterations  With  Early 
Drops  For  Government 
Testing  (Beta) 


Prototype  Development 


Confidence  Demos 
(User  Run  Event) 


Normalized,  quantitative 
defect  metrics  capture  and 
presentation 


Day  in  the  Life  Test 
(Scenario  Based  Testing) 


w 


Award  Fee  Plan 
(quality  driven) 


Government 
Test  Feedback 


Software  Quality  Risk 
Management  Plan 
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Final  Thoughts 

Deploy  Quality 
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“TEST  LIKE  YOU  FLY” 


Summary:  Focus  On  Quality  and  Suitability 


Bake  unscripted  testing  into  the  contract 


►  Include  government  developed  tests  as  quality  gates,  eg.  DitL 

►  Consider  including  deliverable  automated  test  scripts  as  a  part  of  the  contract 

►  Scrutinize  proposed  test  strategy  and  make  sure  that  it  is  appropriate  for  the  system 
being  developed  (requirements  verification  is  seldom  enough  with  today’s  software) 

Risk  Management  Plan  should  provide  a  quantitative  way  to  measure  software  quality  as 
a  positive  trend  and  a  measurable  mitigation  plan  to  address  deficiencies 

►  Award  the  contractor  for  delivering  quality  products.  Avoid  “binary”  criteria  that 
encourage  schedule  over  quality 

•  NOT  GOOD  ENOUGH:  Deliver  a  product  that  meets  requirements 

BETTER:  Deliver  a  product  that  meets  requirements  AND  exhibits  a  high  level  of  quality  and 
suitability  as  qualified  by: 

1  Risk  mitigation  activities  (are  we  effectively  identifying  and  burning  down  risk?) 

2.  Quantitative  metrics  (DDPs,  Hrs/Defect,  Real-time  Dashboard  Views)  for  trends,  not  just  threshold  points 
B.  User  input  from  government  test  and  confidence  demos 

Do  whatever  you  can  to  get  the  user  community  involved  early  -  design  reviews, 
prototypes,  confidence  demos  and  early  drops  for  government  testing 


Consider  iterations  and  early  drops  over  formal  incremental  deliveries.  Making 
something  operational  is  expensive  and  time  consuming 

You  might  (will)  end  up  with  costly  overruns  if  (when)  the  schedule  slips 
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Thank  You 


All  trademarks,  service  marks,  and  trade  names  are 
property  of  their  respective  owners. 
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Acronym  List 


►  AEHF  -  Advanced  Extremely  High  Frequency 

►  CM  -Configuration  Management 

►  CPI  -  Cost  Performance  Index 

DBCM  -  Database  Consistency  Manager 

►  DDP  -  Defect  Density  Profile 

►  DITL  -  Day  in  the  Life 

►  DLL  -  Dynamic  Link  Library 

►  FIT  -  Formal  Integration  Test 

►  HMI  -  Human  Machine  Interface 

►  HW  -  Hardware 

►  IP  -  International  Partner 

►  LDR  -  Low  Data  Rate 

►  MDR  -  Medium  Data  Rate 

►  MCS  -  Mission  Control  Segment 
MFC  -  Microsoft  Foundation  Classes 
MODC  -  Mission  Ops  Data  Capture 
MOPS  -  Mission  Operations  Element 

►  MPE  -  Mission  Planning  Element 


MPSS  -  Mission  Planning  Sub-System 

►  ODBC  -  Open  Database  Connectivity 

►  OSSE  -  Operations  Segment 

Sustainment  Element 

►  OUE  -  Operational  Utility  Event 

►  PM  -  Program  Management 

►  RSSC-  Regional  SATCOM  Support 

Center 

RTO  -  Responsible  Test  Organization 

►  SOC  -  Satellite  Operations  Center 

►  SOM  -  System  Operational  Manager 

►  SPI  -  Schedule  Performance  Index 

►  SQA  -  Software  Quality  Assurance 

►  SQL  -  Structured  Query  Language 

►  SW  -  Software 

UCC  -  Unified  Combat  Control 

►  XDR  -  extended  Data  Rate 


